Refutation Analysis for Constraint Satisfaction Problems

Good heuristics have long been credited as some of the most important components of constraint satisfaction search algorithms. Understanding their behaviour has been the subject of a significant number of both theoretical and empirical research papers, yet from a practical standpoint, questions about certain important performance characteristics remained unanswered. For example, what is the smallest possible refutation of an insoluble search tree? Are heavy-tailed runtime distributions encountered for a given search algorithm inherently heavy-tailed? Refutations oftentimes involve only a small subset of the uninstantiated variables. Could simply reordering those variables lead to better refutations? Are those variables special? This dissertation introduces a methodology for finding some of those answers through a type of empirical analysis never attempted before. At the core of the research presented here lies an algorithm that takes on the computationally expensive task of obtaining optimal refutations for all the insoluble subtrees encountered during search. By studying these optimal refutations, by comparing them to those refutations obtained with standard search algorithms, by analyzing their runtime distributions, as well as through a heatmaps-based visual analysis of the mistakes encountered, one can gain significant insights into the inner working of search, understand the nature of the mistakes made by current algorithms, and the potential and limitations of the heuristics used. Furthermore, the ability to obtain insoluble subproblems that are guaranteed to be difficult can lead to the creation of more sophisticated methods for generating hard benchmark problems. Ultimately, a better understanding of problem hardness and typical-case complexity can empower algorithm designers and problem solvers, leading to the design of better, more robust heuristics.